The present invention relates generally to interface devices for allowing humans to interface with computer systems, and more particularly to computer interface devices that allow the user to provide input to computer systems and provide force feedback to the user.
Computer systems are used extensively to implement many applications, such as word processing, data management, simulations, games, and other tasks. A computer system typically displays a visual environment to a user on a display screen or other visual output device. Users can interact with the displayed environment to perform functions on the computer, play a game, experience a simulated environment, use a computer aided design (CAD) system, etc. One visual environment that is particularly common is a graphical user interface (GUI). GUI's present visual images which describe various graphical metaphors of a program or operating system implemented on the computer. Common GUI's include the Windows™ operating system from Microsoft Corporation and the MacOS operating system from Apple Computer, Inc. The user typically moves a displayed, user-controlled graphical object, such as a cursor or pointer, across a computer screen and onto other displayed graphical objects or predefined screen regions, and then inputs a command to execute a given selection or operation. The objects or regions (“targets”) can include, for example, icons, windows, pull-down menus, buttons, and scroll bars. Most GUI's are currently 2-dimensional as displayed on a computer screen; however, three dimensional (3-D) GUI's that present simulated 3-D environments on a 2-D screen can also be provided. Other programs or environments that may provide user-controlled graphical objects such as a cursor or a “view” controlled by the user include graphical “web pages” or other environments offered on the World Wide Web of the Internet, CAD programs, video games, virtual reality simulations, etc.
The user interaction with and manipulation of the computer environment is achieved using any of a variety of types of human-computer interface devices that are connected to the computer system controlling the displayed environment. In most systems, the computer updates the environment in response to the user's manipulation of a user-manipulatable physical object (“user object”) that is included in the interface device, such as a mouse, joystick, etc. The computer provides feedback to the user utilizing the display screen and, typically, audio speakers.
A computer mouse is a common user object used to interact with a GUI or other graphical environment. A mouse (and other mouse-type devices such as a track ball) is typically used as a position control device in which displacement of the mouse in a planar workspace (e.g. on a mouse pad) is directly correlated to displacement of the user-controlled graphical object, such as a cursor, displayed on the screen. This displacement correlation may not be a one-to-one correspondence, since the cursor position may be scaled according to a constant mapping from the mouse position e.g., the mouse may be moved a distance of one inch on a mouse pad which causes the controlled cursor to move four inches across the screen. In most cases, small movements of the mouse are scaled to large motions of the cursor on the screen to allow the user to easily point to targets in all areas of the screen. The user can typically change the scaling or “pointer speed” of a cursor to a desired level, which is the ratio or scaling factor of cursor movement to mouse movement, using menus provided in the operating system or application program.
The scaled cursor movement in a GUI works well for coarse cursor motion, which is the broad, sweeping motion of the cursor that brings the cursor from one global area on the screen to another. Accuracy of cursor motion is not critical for coarse motion, but speed of the cursor is ideally, the cursor traverses the desired distance on the screen quickly and efficiently. For such tasks, it is valuable for the cursor to move a large distance with small motions of the physical mouse hardware. However, a problem occurs in mouse-type devices when the user wishes to move the cursor a short distance or in small increments (“fine positioning”). For tasks in which accurate positioning of the cursor is needed, such as target acquisition tasks, the large scaling of mouse movement to cursor movement is inadequate or even harmful. For example, the user may wish to move the cursor onto a GUI target such as an icon or menu item. If very small motions of the mouse result in large cursor motion, the user may simply lack the manual dexterity to acquire the target. Certain target acquisition tasks where the targets are very small can be particularly challenging even if the mapping between the cursor and the mouse is reasonable for most other cursor motion activities. For example, in drawing programs it is often required that a user position the cursor on a very small “point” or “node” on the screen; and in some cases, the target can be as small as a single display pixel. For such situations, a scaling that causes large motions of the cursor for small motions of the mouse may make a target acquisition task physically impossible for the user.
Mouse “ballistics” or “ballistic tracking” is typically used to alleviate the scaling problem for fine positioning of the cursor. Ballistics refers to the technique of varying the scaling between motion of a physical mouse and motion of a displayed cursor depending upon the velocity of the mouse in its workspace. The assumption is that if the user is moving the mouse very quickly, the user is likely performing a “coarse motion” task on the screen, and therefore the mouse driver scales small motions of the mouse to large motions of the cursor. Conversely, if the user is moving the mouse very slowly, the user is likely performing a fine positioning task on the screen, and the mouse driver scales small motions of the mouse to small motions of the cursor. Such a variable scaling technique is disclosed in U.S. Pat. No. 4,734,685 of Watanabe and U.S. Pat. No. 5,195,179 of Tokunaga.
Many algorithms can be used for mouse ballistics. The simplest method is to designate a threshold velocity such that if the mouse is moving faster than the threshold velocity, a large scaling of cursor position is made so that small motions of the mouse cause large motions of the cursor; and if the mouse is moving slower than the threshold velocity, a smaller scaling is made so that small motions of the mouse cause small motions of the cursor. A more sophisticated and more common method is to gradually change the scaling in accordance with mouse velocity using a continuous function. This can be a simple linear function, such as a direction relation between mouse speed and the distance the cursor moves for a given increment of mouse motion, or a non-linear function that is optimized in a particular way. The “mapping” of the cursor to the mouse is the method of translating the mouse position in its workspace to a cursor position on the display screen and may involve ballistics or other algorithms and scale factors.
A problem occurs when standard ballistics techniques are used with force feedback interface devices. Force feedback interface devices allow a user to experience forces on the manipulated user object based on interactions and events within the displayed graphical environment. Typically, computer-controlled motors or other actuators are used to output forces on the user object in provided degrees of freedom to simulate various sensations, such as an obstruction force when moving the cursor into a wall, a damping force to resist motion of the cursor, and a spring force to bias the cursor to move back toward a starting position of the spring.
Force feedback devices can be implemented in many forms, such as a joystick, mouse, steering wheel, etc.
When these and other types of forces are implemented in conjunction with mouse ballistics, a conflict occurs between the use of ballistics position and force feedback output. In general, force feedback is generated based directly on motion of the mouse while visual feedback such as movement of the cursor does not correspond directly with motion of the mouse due to scaling and ballistics. As explained above, 1 mm displacement of the mouse may cause different visual results in cursor motion based on the mouse velocity when using ballistics. Thus, when implementing forces in an interface device, motion of the mouse can no longer be consistently correlated to cursor motion on the screen due to the variable scaling. This is a particular problem for a force feedback mouse system because in general, feel sensations such as springs, surfaces, dampers, textures, masses, and other spatially related physical phenomenon rely on a constant, predictable mapping between the motion of the mechanical object (mouse), the forces generated on the mouse, and graphically displayed interactions on the screen (cursor motion).
For example, a spring sensation can be used when the user manipulates a cursor to stretch a graphically displayed element on the screen, such as a line in a drawing program or a window in a GUI. The user positions the cursor on the line, and moves the cursor to stretch the line. Visually, this stretch is displayed based on cursor motion. In addition, the accompanying spring sensation outputs a resistance force that increases linearly with displacement. Traditional force feedback systems would use displacement of the mouse in its workspace as the displacement magnitude required to calculate the spring force. If no ballistics are in effect, no problem exists because the cursor displacement used in the visual display has a consistent, constant mapping to the mouse displacement used in the force “display.” However, if ballistics are used to map physical mouse motion to displayed cursor motion, the motion of the cursor varies depending upon mouse velocity, causing a potential conflict: the stretch displayed visually is based on the variable mapping adjusted by the ballistic algorithm, while the stretch force is based on pure mouse motion. This conflict becomes a problem when, for example, the user stretches a line very quickly in one direction from a starting position (both a mouse starting position and a screen starting position), changes to the opposite direction, and unstretches the line very slowly toward the starting position. The motion in the first direction has a large scaling of mouse motion to cursor motion, while the motion in the second direction has a small scaling of mouse motion to cursor motion. The user may move the mouse in its physical workspace exactly the same distance in both directions (returning to the mouse starting position), but on the screen the line might stretch very far (when moving fast), but then come back only a small distance (when moving slow). Thus, visually, the cursor did not return to the screen starting position.
Using a traditional mouse, this frequently occurs and is not a problem. However, on a force feedback mouse where force display is based on mouse motion and visual display is based on cursor motion, a disconcerting dichotomy is noticed by the user. Since the feel of the stretch is based on mouse motion, if the user stretches in one direction and returns to the starting position, the user feels stretching the line a given displacement and then unstretching the line that same displacement, with an end result of no stretch. But, on the screen where the cursor mapping is based on velocity, the user would see the line stretching far in one direction and then unstretching only a small amount in the opposite direction so that the cursor is not yet back to the starting position. The user would visually expect to still feel some stretched tension, but no such tension exists since the mouse is back at the starting position in its own workspace. Thus a problem is evident in force feedback mouse-type devices: ballistics are needed to allow dexterous cursor control, yet ballistics distorts the seeing-feeling relationship.
Thus, adjusting the mapping between physical mouse motion and displayed cursor motion makes sense for graphical display, but does not make sense for force feedback where physical realism is critical to effective sensation generation. Therefore, for force feedback mouse systems, it would be preferred to eliminate mouse ballistics. Unfortunately, such a force feedback mouse would not be optimized for both fine positioning and coarse motion, as is true of traditional mice.
In addition, mouse ballistics causes another problem that causes difficulty in force feedback mouse implementation. As described above, moving the mouse in one direction quickly and then moving it back in the other direction slowly creates a situation where the mouse hardware has returned to its starting position but the cursor may be far away from its starting position. This illustrates that the frame of the cursor and the frame of the mouse have shifted or become offset.
If this offset becomes too large, the user may not be able to reach some parts of the screen within the range of motion of the mouse. In a typical mouse, the offset is corrected through a process called “indexing.” Indexing is achieved in a typical mouse by lifting the mouse off the table and repositioning it after the mouse has hit a limit, while the cursor remains fixed in position. This brings the mouse and the cursor frames back to a smaller, more comfortable offset. A force feedback mouse may have a limited workspace due to cost constraints and may not be able to be lifted off the table and repositioned. In addition, the mouse hitting a physical limit to its workspace is disconcerting for a user expecting realistic force feedback. Thus, traditional indexing may not be practical. However, since ballistics needs indexing to restore the frame offsets, and since ballistics and indexing are both traditional mouse techniques that conflict with typical force feedback functionality, a solution is needed that reconciles both the ballistics and the indexing problem in force feedback hardware.